spike(rust/decode): FFmpeg vs gstreamer-rs binding decision (#108) by natashaannn · Pull Request #109 · ragTechDev/deckcreate

natashaannn · 2026-07-17T14:49:23Z

Summary

Implements the issue #108 spike: a minimal Rust harness at spike/rust-decode-spike/ comparing ffmpeg-the-third vs gstreamer-rs as the decode/encode binding for the RFC 0001 native render engine.

Mac M3 results are real and verified in this branch (not just scaffolded) — see FINDINGS.md for the full evidence per criterion. Headlines:

ffmpeg-the-third: pixel-exact frame-accurate seek+decode on both software and VideoToolbox hardware paths. No high-level hwaccel API exists in the crate — required ~40 lines of hand-written unsafe FFI (documented as a working reference for Epic 1).
gstreamer-rs: hardware selection is just naming vtdec_hw instead of avdec_h264 (zero unsafe code), and it does negotiate/preroll successfully — but its pixel output is outside tolerance (confirmed not an off-by-one-frame bug by diffing against neighboring frames), likely a GL-readback colorimetry issue, plus a GStreamer-GL-WARNING about needing an NSApplication run loop that a render-engine binary won't have.
Criterion Feat/video editor #5 (licensing) turned out to apply to both candidates, not just gstreamer-rs as originally scoped: this machine's Homebrew FFmpeg is --enable-gpl, so gstreamer-rs's software fallback (avdec_h264, via gst-libav) is GPL-contaminated, while vtdec_hw (Apple system frameworks only) is clean. Flagged as its own prerequisite issue for Epic 1 regardless of binding choice.

Recommendation: ffmpeg-the-third, contingent on the Windows/NVIDIA row below. Full rationale in FINDINGS.md.

Also corrected issue #108's spikes/ (plural) path to spike/ (singular) to match the established convention from the audio-sync spike (#93–#99) — standalone Cargo.toml per spike, no root-level workspace.

What's left for reviewers

Mac M2 and Windows+NVIDIA rows in FINDINGS.md are open — @Saloni @victoria, whichever of you has that hardware, please run the commands in spike/rust-decode-spike/README.md and paste your output into FINDINGS.md (or a comment here). The Windows candidates also need a nvh264dec/NVDEC code path added — TODOs are left in both main.rs files marking exactly where.

Test plan

npm test — 20/20 suites pass (this change doesn't touch JS/TS)
All 5 implementation-guide status checks pass (docs/implementation-guides/issue-108-rust-decode-spike.md)
cargo test passes for both candidate crates (PPM/pixel-compare unit tests)
Both candidates verified end-to-end against committed fixtures on Apple M3 (software + hardware paths)
Mac M2 verification (open — needs contributor with that hardware)
Windows + NVIDIA verification (open — needs contributor with that hardware, plus the NVDEC code path)

🤖 Generated with Claude Code

…fixtures Issue #108 needs a real .mp4 to test seek/decode accuracy for the ffmpeg-the-third vs gstreamer-rs binding decision. testsrc2 content is reproducible per-frame-index, so the reference PPMs stay valid ground truth even if the clip is regenerated later. Follows the spike/<name>/ convention from #93-99 rather than the issue's original spikes/ (plural) text, which has been corrected in the issue body. AC: issue #108 spike scope (fixture prerequisite for all other criteria)

… software + VideoToolbox Answers criteria #1 and #2 from issue #108 for this candidate. Software path is a straight port of the standard ffmpeg-next seek-then-decode- forward pattern; the --hw path required hand-written unsafe FFI through ffmpeg_the_third::ffi (av_hwdevice_ctx_create + a get_format callback on the *unopened* AVCodecContext + av_hwframe_transfer_data) because the crate has no high-level hwaccel wrapper at all. Verified on this M3 Mac: both paths are pixel-exact against the ffmpeg-CLI reference frames, and the hardware path is confirmed to actually engage VideoToolbox rather than silently falling back to software. AC: issue #108 criteria #1 (hardware codec access), #2 (seek/decode accuracy) for the ffmpeg-the-third candidate

…software + VideoToolbox Answers criteria #1 and #2 from issue #108 for the gstreamer-rs candidate. Hardware selection is just naming a different pipeline element (vtdec_hw vs avdec_h264) instead of ffmpeg-the-third's raw FFI hwaccel setup - a real build-complexity difference between the two candidates worth weighing on its own. Verified on this M3 Mac: the software path (avdec_h264) is a close but not pixel-exact match to the ffmpeg-CLI reference (mean diff 1.4, within tolerance). vtdec_hw negotiates and prerolls successfully (hardware path genuinely active, confirmed by diffing the decoded frame against its timestamp neighbors to rule out an off-by-one-frame seek bug instead), but its output pixel-mismatches the reference well outside tolerance (mean diff 3.9, max 168/255) - most likely a colorimetry/GL-readback difference given a plain `vtdec_hw ! videoconvert` pipeline outputs GLMemory-backed buffers. Also surfaced a `GStreamer-GL-WARNING` that vtdec_hw's GL path expects an NSApplication run loop on macOS, which a render-engine binary won't have by default. Left as an honest FAIL rather than a loosened tolerance or a deeper GL-pipeline fix - this friction is exactly what the spike exists to surface for criterion #1. AC: issue #108 criteria #1 (hardware codec access), #2 (seek/decode accuracy) for the gstreamer-rs candidate

Records the actual evidence gathered on this M3 Mac against all five evaluation criteria in issue #108, including two findings beyond what either candidate's happy-path docs would suggest: gstreamer-rs's vtdec_hw negotiates hardware decode but produces pixel output outside tolerance (root-cause not yet fixed, ruled out an off-by-one-frame seek bug specifically so this isn't mistaken for one), and the GPL-vs-LGPL FFmpeg build question applies equally to both candidates, not just gstreamer-rs as the issue originally scoped it. Recommends ffmpeg-the-third, contingent on the still-open Windows/NVIDIA row. README gives the exact per-platform setup and run commands for the Mac M2 and Windows/NVIDIA contributors to fill in the remaining rows. AC: issue #108 all five evaluation criteria; FINDINGS.md deliverable

Future agents touching RFC 0001 or issue #108 need to know spike/ holds throwaway crates separate from the numbered refactor phases, and where the decode-binding spike's findings live.

natashaannn · 2026-07-17T14:53:58Z

Ready-to-paste Claude Code prompts for the remaining hardware rows

Copy the block matching your machine into a Claude Code session in this repo. Each one is
self-contained — it explains what's already done and what's expected back.

Mac M2

This repo has a spike branch `spike/issue-108-decode-binding` (PR #109, issue #108) comparing
`ffmpeg-the-third` vs `gstreamer-rs` for hardware-accelerated video decode. Real Mac M3 results
are already recorded in `spike/rust-decode-spike/FINDINGS.md`. I'm on a Mac M2 — fill in the Mac
M2 rows in that file with real results from this machine, not scaffolding.

1. Checkout `spike/issue-108-decode-binding` and pull latest.
2. Read `spike/rust-decode-spike/README.md` for prerequisites (likely
   `brew install pkgconf ffmpeg gstreamer`) and install anything missing.
3. Read `spike/rust-decode-spike/FINDINGS.md` in full first so you match its existing level of
   detail and format — don't invent a new one.
4. Run both candidates against both fixtures, software and `--hw`, exactly as documented in the
   README (`ffmpeg-candidate` and `gstreamer-candidate`, timestamps 0.5 and 1.5, both reference
   PPMs), plus `cargo test` for both crates.
5. If anything differs from the M3 results already recorded — VideoToolbox doesn't engage,
   different pixel diffs, different build friction — investigate briefly like the M3 findings did.
   Don't just report pass/fail; explain what you saw. Also re-check the criterion #5 licensing
   table (`gst-inspect-1.0 <plugin> | grep License`, `otool -L` on the linked dylibs) and confirm
   it matches the M3 findings on this machine.
6. Update the Mac M2 cells in `spike/rust-decode-spike/FINDINGS.md` with your actual results, and
   update "Recommendation" only if your results actually change the conclusion.
7. Commit with a `docs(spike):` prefix and push to the existing branch
   `spike/issue-108-decode-binding` — don't open a new branch or PR, #109 already exists.
8. Comment on PR #109 summarizing what you found.

Don't modify the candidate source code unless something doesn't build on your machine — this is a
verification pass, not a rewrite.

Windows + NVIDIA

This repo has a spike branch `spike/issue-108-decode-binding` (PR #109, issue #108) comparing
`ffmpeg-the-third` vs `gstreamer-rs` for hardware-accelerated video decode. Real Mac M3 results
are already recorded in `spike/rust-decode-spike/FINDINGS.md`. I'm on Windows with an NVIDIA GPU —
the NVENC/NVDEC hardware path isn't implemented yet for either candidate (left as a TODO), so you
need to implement it, then fill in the Windows+NVIDIA rows with real results, not scaffolding.

1. Checkout `spike/issue-108-decode-binding` and pull latest.
2. Read `spike/rust-decode-spike/README.md` and `FINDINGS.md` in full, and read both
   `spike/rust-decode-spike/ffmpeg-candidate/src/main.rs` and
   `spike/rust-decode-spike/gstreamer-candidate/src/main.rs` — you're extending the existing
   pattern, not starting over.
3. Install prerequisites: FFmpeg dev libs + pkg-config (e.g. via vcpkg), and GStreamer's official
   Windows installer (development package) including the `nvcodec` plugin.
4. `ffmpeg-candidate`: the `hw` module currently only has a `#[cfg(target_os = "macos")]`
   VideoToolbox implementation. Add a `#[cfg(target_os = "windows")]` variant that uses
   `AVHWDeviceType::CUDA` instead of `AVHWDeviceType::VIDEOTOOLBOX` in `av_hwdevice_ctx_create`,
   with `get_format` looking for `AVPixelFormat::CUDA`. Wire it into the existing `--hw` flag the
   same way the macOS path is wired in. Leave the seek/decode/compare flow unchanged.
5. `gstreamer-candidate`: currently hardcodes `vtdec_hw` for `--hw`. Make the decoder-element
   choice platform-conditional: `nvh264dec` (from the `nvcodec` plugin) on Windows, `vtdec_hw` on
   macOS.
6. Build and run both candidates against both fixtures, software and `--hw`, plus `cargo test` for
   both crates.
7. Investigate honestly like the M3 findings did — if hardware decode fails to engage, produces
   wrong pixels, or the build has friction, document exactly what happened, not just pass/fail.
   Check plugin licensing for criterion #5 too (`gst-inspect-1.0 nvcodec | grep License`, and what
   the nvcodec DLL links against) and add a Windows/NVIDIA row to that table.
8. Update the Windows+NVIDIA rows in `spike/rust-decode-spike/FINDINGS.md`, and update
   "Recommendation" if your results change the conclusion — the current recommendation is
   explicitly contingent on this row, so it's entirely possible it does.
9. Commit code changes separately from findings (code as `spike(rust/ffmpeg):` /
   `spike(rust/gstreamer):`, findings as `docs(spike):`), and push to the existing branch
   `spike/issue-108-decode-binding`.
10. Comment on PR #109 summarizing what you found and whether the recommendation should change.

Apple M2 (MacBook Air Mac14,2, macOS 14.5, Rust 1.97.1) results match M3 exactly across all eight test cases: - ffmpeg-the-third software: pixel-exact (mean=0, max=0) both timestamps - ffmpeg-the-third --hw (VideoToolbox): pixel-exact, HARDWARE PATH active - gstreamer-rs software: mean≈1.39, max=6, PASS (matches M3) - gstreamer-rs --hw (vtdec_hw): mean≈3.86, max≈175, FAIL (matches M3) - cargo test: 3/3 ffmpeg-candidate, 2/2 gstreamer-candidate One notable difference from M3: GStreamer-GL-WARNING about NSApplication did not appear on macOS 14.5 (Darwin 23.5.0). vtdec_hw pixel failure is identical in magnitude regardless, confirming the colorimetry issue is not GL-warning-correlated. Criterion #5 licensing confirmed matching M3: applemedia/vtdec_hw is LGPL + Apple system frameworks only (clean); libgstlibav links this machine's Homebrew ffmpeg 8.1.2 (--enable-gpl, same risk as M3). Recommendation unchanged: ffmpeg-the-third. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

missabawse · 2026-07-19T15:27:42Z

Mac M2 verification complete

Hardware: Apple M2 (MacBook Air, Mac14,2), macOS 14.5, Rust 1.97.1

All results match M3 exactly across all 8 test cases:

Candidate / path	t=0.5s	t=1.5s
`ffmpeg-the-third`, software	mean=0.000, max=0 — PASS (pixel-exact)	mean=0.000, max=0 — PASS (pixel-exact)
`ffmpeg-the-third`, `--hw` (VideoToolbox)	`HARDWARE PATH: active`; mean=0.000, max=0 — PASS (pixel-exact)	`HARDWARE PATH: active`; mean=0.000, max=0 — PASS (pixel-exact)
`gstreamer-rs`, software	mean=1.391, max=6 — PASS	mean=1.377, max=6 — PASS
`gstreamer-rs`, `--hw` (`vtdec_hw`)	`HARDWARE PATH: active`; mean=3.855, max=168 — FAIL	`HARDWARE PATH: active`; mean=3.886, max=181 — FAIL

cargo test: 3/3 ffmpeg-candidate, 2/2 gstreamer-candidate — all pass.

One notable difference from M3: The GStreamer-GL-WARNING: An NSApplication needs to be running on the main thread did not appear on macOS 14.5 (Darwin 23.5.0). The vtdec_hw pixel failure is the same magnitude regardless — the colorimetry issue is not GL-warning-correlated but a consistent vtdec_hw characteristic across both M-series chips.

Criterion #5 licensing (M2): matches M3 findings exactly — applemedia/vtdec_hw is LGPL + Apple system frameworks only (clean); libgstlibav links this machine's Homebrew ffmpeg 8.1.2 (--enable-gpl), same risk profile as M3.

Build notes: Rust not preinstalled (rustup install required). pkgconf and gstreamer installed via Homebrew before the first build. No build errors or patching required on either candidate. First-build times ~17s each from cold.

Recommendation unchanged: ffmpeg-the-third. The M2 results add confidence — VideoToolbox pixel-exact behavior is consistent across both M-series targets, not an M3-specific result.

FINDINGS.md updated and pushed to this branch.

natashaannn added 5 commits July 17, 2026 22:30

docs(claude-md): note rust-decode-spike location and status

6dbf61f

Future agents touching RFC 0001 or issue #108 need to know spike/ holds throwaway crates separate from the numbered refactor phases, and where the decode-binding spike's findings live.

natashaannn requested review from missabawse and victoria-lo July 17, 2026 15:22

victoria-lo and others added 4 commits July 19, 2026 11:03

spike(rust/ffmpeg): add windows cuda hw decode path

c919fa3

spike(rust/gstreamer): select hw decoder by platform

43008b4

docs(spike): add windows nvidia decode results and recommendation update

676bb82

natashaannn merged commit 443e8b7 into main Jul 20, 2026
1 check passed

natashaannn deleted the spike/issue-108-decode-binding branch July 20, 2026 03:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

spike(rust/decode): FFmpeg vs gstreamer-rs binding decision (#108) - #109

spike(rust/decode): FFmpeg vs gstreamer-rs binding decision (#108)#109
natashaannn merged 9 commits into
mainfrom
spike/issue-108-decode-binding

natashaannn commented Jul 17, 2026

Uh oh!

natashaannn commented Jul 17, 2026

Uh oh!

missabawse commented Jul 19, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

natashaannn commented Jul 17, 2026

Summary

What's left for reviewers

Test plan

Uh oh!

natashaannn commented Jul 17, 2026

Ready-to-paste Claude Code prompts for the remaining hardware rows

Mac M2

Windows + NVIDIA

Uh oh!

missabawse commented Jul 19, 2026

Mac M2 verification complete

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants